Sorting algorithm

In computer science and mathematics, a sorting algorithm is an algorithm that puts elements of a list in a certain order. The most-used orders are numerical order and lexicographical order. Efficient sorting is important to optimizing the use of other algorithms (such as search and merge algorithms) that require sorted lists to work correctly; it is also often useful for canonicalizing data and for producing human-readable output. More formally, the output must satisfy two conditions:

  1. The output is in nondecreasing order (each element is no smaller than the previous element according to the desired total order);
  2. The output is a permutation, or reordering, of the input.

Since the dawn of computing, the sorting problem has attracted a great deal of research, perhaps due to the complexity of solving it efficiently despite its simple, familiar statement. For example, bubble sort was analyzed as early as 1956.[1] Although many consider it a solved problem, useful new sorting algorithms are still being invented (for example, library sort was first published in 2004). Sorting algorithms are prevalent in introductory computer science classes, where the abundance of algorithms for the problem provides a gentle introduction to a variety of core algorithm concepts, such as big O notation, divide and conquer algorithms, data structures, randomized algorithms, best, worst and average case analysis, time-space tradeoffs, and lower bounds.

Contents

Classification

Sorting algorithms used in computer science are often classified by:

Stability

Stable sorting algorithms maintain the relative order of records with equal keys. If all keys are different then this distinction is not necessary. But if there are equal keys, then a sorting algorithm is stable if whenever there are two records (let's say R and S) with the same key, and R appears before S in the original list, then R will always appear before S in the sorted list. When equal elements are indistinguishable, such as with integers, or more generally, any data where the entire element is the key, stability is not an issue. However, assume that the following pairs of numbers are to be sorted by their first component:

(4, 2)  (3, 7)  (3, 1)  (5, 6)

In this case, two different results are possible, one which maintains the relative order of records with equal keys, and one which does not:

(3, 7)  (3, 1)  (4, 2)  (5, 6)   (order maintained)
(3, 1)  (3, 7)  (4, 2)  (5, 6)   (order changed)

Unstable sorting algorithms may change the relative order of records with equal keys, but stable sorting algorithms never do so. Unstable sorting algorithms can be specially implemented to be stable. One way of doing this is to artificially extend the key comparison, so that comparisons between two objects with otherwise equal keys are decided using the order of the entries in the original data order as a tie-breaker. Remembering this order, however, often involves an additional computational cost.

Sorting based on a primary, secondary, tertiary, etc. sort key can be done by any sorting method, taking all sort keys into account in comparisons (in other words, using a single composite sort key). If a sorting method is stable, it is also possible to sort multiple times, each time with one sort key. In that case the keys need to be applied in order of increasing priority.

Example: sorting pairs of numbers as above by first, then second component:

(4, 2)  (3, 7)  (3, 1)  (5, 6) (original)

(3, 1)  (4, 2)  (5, 6)  (3, 7) (after sorting by second component)
(3, 1)  (3, 7)  (4, 2)  (5, 6) (after sorting by first component)

On the other hand:

(3, 7)  (3, 1)  (4, 2)  (5, 6) (after sorting by first component)
(3, 1)  (4, 2)  (5, 6)  (3, 7) (after sorting by second component, 
                                order by first component is disrupted).

Comparison of algorithms

The complexity of different algorithms in a specific situation.

In this table, n is the number of records to be sorted. The columns "Average" and "Worst" give the time complexity in each case, under the assumption that the length of each key is constant, and that therefore all comparisons, swaps, and other needed operations can proceed in constant time. "Memory" denotes the amount of auxiliary storage needed beyond that used by the list itself, under the same assumption. These are all comparison sorts. The run-time and the memory of algorithms could be measured using various notations like theta, sigma, Big-O, small-o, etc. The memory and the run-times below are applicable for all the 5 notations.

Name Average Worst Memory Stable Method Other notes
Timsort \mathcal{} {n \log n} \mathcal{} {n \log n} \mathcal{} n Yes Insertion & Merging \mathcal{} {n} comparisons when the data is already sorted.
Bubble sort \mathcal{} n^2 \mathcal{} n^2 \mathcal{} {1} Yes Exchanging Tiny code
Cocktail sort — \mathcal{} n^2 \mathcal{} {1} Yes Exchanging
Comb sort — — \mathcal{} {1} No Exchanging Small code size
Gnome sort — \mathcal{} n^2 \mathcal{} {1} Yes Exchanging Tiny code size
Selection sort \mathcal{} n^2 \mathcal{} n^2 \mathcal{} {1} Depends Selection Its stability depends on the implementation. Used to sort this table in Safari or other Webkit web browser [2].
Insertion sort \mathcal{} n^2 \mathcal{} n^2 \mathcal{} {1} Yes Insertion Average case is also \mathcal{O}\left( {n + d} \right), where d is the number of inversions
Shell sort — \mathcal{} {n \log^2 n} \mathcal{} {1} No Insertion
Binary tree sort \mathcal{} {n \log n} \mathcal{} {n \log n} \mathcal{} n Yes Insertion When using a self-balancing binary search tree
Library sort \mathcal{} {n \log n} \mathcal{} n^2 \mathcal{} n Yes Insertion
Merge sort \mathcal{} {n \log n} \mathcal{} {n \log n} \mathcal{} n Yes Merging Used to sort this table in Firefox [3].
In-place merge sort \mathcal{} {n \log n} \mathcal{} {n \log n} \mathcal{} {1} Depends Merging Example implementation here: [4]; can be implemented as a stable sort based on stable in-place merging: [5]
Heapsort \mathcal{} {n \log n} \mathcal{} {n \log n} \mathcal{} {1} No Selection
Smoothsort — \mathcal{} {n \log n} \mathcal{} {1} No Selection An adaptive sort - \mathcal{} {n} comparisons when the data is already sorted, and \mathcal{O}\left( {0} \right) swaps.
Quicksort \mathcal{} n \log n \mathcal{} n^2 \mathcal{} \log n Depends Partitioning Can be implemented as a stable sort depending on how the pivot is handled. Naïve variants use \mathcal{O} \left( n \right) space
Introsort \mathcal{} n \log n \mathcal{} n \log n \mathcal{} \log n No Hybrid Used in SGI STL implementations
Patience sorting — \mathcal{} n \log n \mathcal{} n No Insertion & Selection Finds all the longest increasing subsequences within O(n log n)
Strand sort \mathcal{} n \log n \mathcal{} n^2 \mathcal{} n Yes Selection
Tournament sort \mathcal{} n \log n \mathcal{} n \log n Selection

The following table describes sorting algorithms that are not comparison sorts. As such, they are not limited by a \Omega\left( {n \log n} \right) lower bound. Complexities below are in terms of n, the number of items to be sorted, k, the size of each key, and d, the digit size used by the implementation. Many of them are based on the assumption that the key size is large enough that all entries have unique key values, and hence that n << 2k, where << means "much less than."

Name Average Worst Memory Stable n << 2k Notes
Pigeonhole sort \;n + 2^k \;n + 2^k \;2^k Yes Yes
Bucket sort \;n \cdot k \;n^2 \cdot k \;n \cdot k Yes No Assumes uniform distribution of elements from the domain in the array.
Counting sort \;n+k \;n+k \;n+2^k Yes Yes If k = \mathcal{O}\left( {n} \right) then Avg RT = \mathcal{O}\left( {n} \right)
LSD Radix sort \;n \cdot \frac{k}{d} \;n \cdot \frac{k}{d} \mathcal{} n Yes No
MSD Radix sort \;n \cdot \frac{k}{d} \;n \cdot \frac{k}{d} \mathcal{} n + \frac{k}{d} \cdot 2^d Yes No Stable version uses an external array of size n to hold all of the bins
MSD Radix sort \;n \cdot \frac{k}{d} \;n \cdot \frac{k}{d} \frac{k}{d} \cdot 2^d No No In-Place. k / d recursion levels, 2d for count array
Spreadsort \;n \cdot \frac{k}{d} \;n \cdot \left( {\frac{k}{s} + d} \right) \;\frac{k}{d} \cdot 2^d No No Asymptotics are based on the assumption that n << 2k, but the algorithm does not require this.

The following table describes some sorting algorithms that are impractical for real-life use due to extremely poor performance or a requirement for specialized hardware.

Name Average Worst Memory Stable Comparison Other notes
Bead sort N/A N/A — N/A No Requires specialized hardware
Simple pancake sort \mathcal{} n \mathcal{} n \mathcal{} {\log n} No Yes Count is number of flips.
Sorting networks \mathcal{} {\log n} \mathcal{} {\log n} \mathcal{} {n \cdot \log (n)} Yes No Requires a custom circuit of size \mathcal{O}\left( n \cdot \log (n) \right)

Additionally, theoretical computer scientists have detailed other sorting algorithms that provide better than \mathcal{O}\left( {n \log n} \right) time complexity with additional constraints, including:

Summaries of popular sorting algorithms

Bubble sort

A bubble sort, a sorting algorithm that continuously steps through a list, swapping items until they appear in the correct order.
Main article: Bubble sort

Bubble sort is a straightforward and simplistic method of sorting data that is used in computer science education. The algorithm starts at the beginning of the data set. It compares the first two elements, and if the first is greater than the second, then it swaps them. It continues doing this for each pair of adjacent elements to the end of the data set. It then starts again with the first two elements, repeating until no swaps have occurred on the last pass. This algorithm is highly inefficient, and is rarely used, except as a simplistic example. For example, if we have 100 elements then the total number of comparisons will be 10000. A slightly better variant, cocktail sort, works by inverting the ordering criteria and the pass direction on alternating passes. The modified Bubble sort will stop 1 shorter each time through the loop, so the total number of comparisons for 100 elements will be 4950.

Bubble sort average case and worst case are both O(n²).

Insertion sort

Main article: Insertion sort

Insertion sort is a simple sorting algorithm that is relatively efficient for small lists and mostly-sorted lists, and often is used as part of more sophisticated algorithms. It works by taking elements from the list one by one and inserting them in their correct position into a new sorted list. In arrays, the new list and the remaining elements can share the array's space, but insertion is expensive, requiring shifting all following elements over by one. Shell sort (see below) is a variant of insertion sort that is more efficient for larger lists.

Shell sort

Shell sort was invented by Donald Shell in 1959. It improves upon bubble sort and insertion sort by moving out of order elements more than one position at a time. One implementation can be described as arranging the data sequence in a two-dimensional array and then sorting the columns of the array using insertion sort.

Merge sort

Main article: Merge sort

Merge sort takes advantage of the ease of merging already sorted lists into a new sorted list. It starts by comparing every two elements (i.e., 1 with 2, then 3 with 4...) and swapping them if the first should come after the second. It then merges each of the resulting lists of two into lists of four, then merges those lists of four, and so on; until at last two lists are merged into the final sorted list. Of the algorithms described here, this is the first that scales well to very large lists, because its worst-case running time is O(n log n). Merge sort has seen a relatively recent surge in popularity for practical implementations, being used for the standard sort routine in the programming languages Perl[5], Python (as timsort[6]), and Java (also uses timsort as of JDK7[7]), among others. Merge sort has been used in Java at least since 2000 in JDK1.3.[8][9]

Heapsort

Main article: Heapsort

Heapsort is a much more efficient version of selection sort. It also works by determining the largest (or smallest) element of the list, placing that at the end (or beginning) of the list, then continuing with the rest of the list, but accomplishes this task efficiently by using a data structure called a heap, a special type of binary tree. Once the data list has been made into a heap, the root node is guaranteed to be the largest(or smallest) element. When it is removed and placed at the end of the list, the heap is rearranged so the largest element remaining moves to the root. Using the heap, finding the next largest element takes O(log n) time, instead of O(n) for a linear scan as in simple selection sort. This allows Heapsort to run in O(n log n) time, and this is also the worst case complexity.

Quicksort

Main article: Quicksort

Quicksort is a divide and conquer algorithm which relies on a partition operation: to partition an array, we choose an element, called a pivot, move all smaller elements before the pivot, and move all greater elements after it. This can be done efficiently in linear time and in-place. We then recursively sort the lesser and greater sublists. Efficient implementations of quicksort (with in-place partitioning) are typically unstable sorts and somewhat complex, but are among the fastest sorting algorithms in practice. Together with its modest O(log n) space usage, this makes quicksort one of the most popular sorting algorithms, available in many standard libraries. The most complex issue in quicksort is choosing a good pivot element; consistently poor choices of pivots can result in drastically slower O(n²) performance, but if at each step we choose the median as the pivot then it works in O(n log n). Finding the median, however, is an O(n) operation on unsorted lists, and therefore exacts its own penalty.

Counting Sort

Counting sort is applicable when each input is known to belong to a particular set, S, of possibilities. The algorithm runs in O(|S| + n) time and O(|S|) memory where n is the length of the input. It works by creating an integer array of size |S| and using the ith bin to count the occurrences of the ith member of S in the input. Each input is then counted by incrementing the value of its corresponding bin. Afterward, the counting array is looped through to arrange all of the inputs in order. This sorting algorithm cannot often be used because S needs to be reasonably small for it to be efficient, but the algorithm is extremely fast and demonstrates great asymptotic behavior as n increases. It also can be modified to provide stable behavior.

Bucket sort

Bucket sort is a divide and conquer sorting algorithm that generalizes Counting sort by partitioning an array into a finite number of buckets. Each bucket is then sorted individually, either using a different sorting algorithm, or by recursively applying the bucket sorting algorithm. Thus this is most effective on data whose values are limited (e.g. a sort of a million integers ranging from 1 to 1000). A variation of this method called the single buffered count sort is faster than quicksort and takes about the same time to run on any set of data.

Radix sort

Radix sort is an algorithm that sorts numbers by processing individual digits. n numbers consisting of k digits each are sorted in O(n · k) time. Radix sort can either process digits of each number starting from the least significant digit (LSD) or the most significant digit (MSD). The LSD algorithm first sorts the list by the least significant digit while preserving their relative order using a stable sort. Then it sorts them by the next digit, and so on from the least significant to the most significant, ending up with a sorted list. While the LSD radix sort requires the use of a stable sort, the MSD radix sort algorithm does not (unless stable sorting is desired). In-place MSD radix sort is not stable. It is common for the counting sort algorithm to be used internally by the radix sort. Hybrid sorting approach, such as using insertion sort for small bins improves performance of radix sort significantly.

Distribution sort

Distribution sort refers to any sorting algorithm where data is distributed from its input to multiple intermediate structures which are then gathered and placed on the output. See Bucket sort.

Memory usage patterns and index sorting

When the size of the array to be sorted approaches or exceeds the available primary memory, so that (much slower) disk or swap space must be employed, the memory usage pattern of a sorting algorithm becomes important, and an algorithm that might have been fairly efficient when the array fit easily in RAM may become impractical. In this scenario, the total number of comparisons becomes (relatively) less important, and the number of times sections of memory must be copied or swapped to and from the disk can dominate the performance characteristics of an algorithm. Thus, the number of passes and the localization of comparisons can be more important than the raw number of comparisons, since comparisons of nearby elements to one another happen at system bus speed (or, with caching, even at CPU speed), which, compared to disk speed, is virtually instantaneous.

For example, the popular recursive quicksort algorithm provides quite reasonable performance with adequate RAM, but due to the recursive way that it copies portions of the array it becomes much less practical when the array does not fit in RAM, because it may cause a number of slow copy or move operations to and from disk. In that scenario, another algorithm may be preferable even if it requires more total comparisons.

One way to work around this problem, which works well when complex records (such as in a relational database) are being sorted by a relatively small key field, is to create an index into the array and then sort the index, rather than the entire array. (A sorted version of the entire array can then be produced with one pass, reading from the index, but often even that is unnecessary, as having the sorted index is adequate.) Because the index is much smaller than the entire array, it may fit easily in memory where the entire array would not, effectively eliminating the disk-swapping problem. This procedure is sometimes called "tag sort".[10]

Another technique for overcoming the memory-size problem is to combine two algorithms in a way that takes advantages of the strength of each to improve overall performance. For instance, the array might be subdivided into chunks of a size that will fit easily in RAM (say, a few thousand elements), the chunks sorted using an efficient algorithm (such as quicksort or heapsort), and the results merged as per mergesort. This is less efficient than just doing mergesort in the first place, but it requires less physical RAM (to be practical) than a full quicksort on the whole array.

Techniques can also be combined. For sorting very large sets of data that vastly exceed system memory, even the index may need to be sorted using an algorithm or combination of algorithms designed to perform reasonably with virtual memory, i.e., to reduce the amount of swapping required.

Inefficient/humorous sorts

These are algorithms that are extremely slow compared to those discussed above — Bogosort O(n\cdot n!), Stooge sort O(n^{2.7}).

See also

References

  1. ↑ Demuth, H. Electronic Data Sorting. PhD thesis, Stanford University, 1956.
  2. ↑ Y. Han. Deterministic sorting in \mathcal{O}\left( {n \log \log n} \right) time and linear space. Proceedings of the thiry-fourth annual ACM symposium on Theory of computing, Montreal, Quebec, Canada, 2002,p.602-608.
  3. ↑ M. Thorup. Randomized Sorting in \mathcal{O}\left( {n \log \log n} \right) Time and Linear Space Using Addition, Shift, and Bit-wise Boolean Operations. Journal of Algorithms, Volume 42, Number 2, February 2002 , pp. 205-230(26)
  4. ↑ Han, Y. and Thorup, M. 2002. Integer Sorting in \mathcal{O}\left( {n \sqrt{\log \log n}} \right) Expected Time and Linear Space. In Proceedings of the 43rd Symposium on Foundations of Computer Science (November 16–19, 2002). FOCS. IEEE Computer Society, Washington, DC, 135-144.
  5. ↑ Perl sort documentation
  6. ↑ Tim Peters's original description of timsort
  7. ↑ [1]
  8. ↑ Merge sort in Java 1.3, Sun.
  9. ↑ Java 1.3 live since 2000
  10. ↑ Definition of "tag sort" according to PC Magazine

External links

Sorting algorithms
Theory
Computational complexity theory · Big O notation · Total order · Lists · Stability · Comparison sort
Exchange sorts
Bubble sort · Cocktail sort · Odd-even sort · Comb sort · Gnome sort · Quicksort
Selection sorts
Selection sort · Heapsort · Smoothsort · Cartesian tree sort · Tournament sort · Cycle sort
Insertion sorts
Insertion sort · Shell sort · Tree sort · Library sort · Patience sorting
Merge sorts
Merge sort · Polyphase merge sort · Strand sort · Timsort
Non-comparison sorts
Bead sort · Bucket sort · Burstsort · Counting sort · Pigeonhole sort · Proxmap sort · Radix sort
Others
Topological sorting · Sorting network · Bitonic sorter · Batcher odd-even mergesort · Pancake sorting
Ineffective/humorous sorts
Bogosort · Stooge sort